Field of Art
The disclosure generally relates to the field of digital image and video processing, and more particularly to a color filter array scaler, temporal and spatial video noise reduction, the prevention of highlight clipping in video, and video global tone mapping.
Description of the Related Art
Cameras (and other imaging devices) capture images and videos using one or more lenses which gather light to be focused onto an image sensor. The image sensor is comprised of many individual subpixels. Each subpixel is comprised of a photosite and registers an intensity (luminance) value depending on how many photons are detected by the photosite. To produce a color image, the individual subpixels are designed to be sensitive to only certain wavelengths of the visible spectrum, for instance, to red, green, or blue portions of the visible spectrum. This selective sensitivity may be achieved by the use of a filter. Combinations of red, green, and blue subpixels are used to create a color pixel for a final image that is presented to the user. As the subpixels are arranged in a grid pattern on the image sensor, a typical arrangement for the subpixels is to have a repeating square of 2×2 subpixels, with each 2×2 square having one red, one blue, and two green subpixels. A Bayer color filter array includes color filters arranged in such a configuration, and is used widely in many cameras. Raw data captured using a Bayer color filter array includes individual red, blue, and green intensity values, and are processed to generate individual pixels that combine the red, blue, and green subpixel values together. These subpixel color values represent the intensity level of a respective color for that pixel. Each subpixel color value is encoded numerically, and has a range depending upon the bit depth of the image format. In some cases, the subpixel values of an image are adjusted to compensate for lightning irregularities.
As noted above, image sensors detect the intensity of light incident upon the image sensor, and converts these various light intensities to an analog signal (e.g., a voltage). This conversion process generates a certain amount of noise, and the amount of noise may change depending on the hardware configuration of the image sensor (e.g., larger photosites may generate lower noise). While the noise generated by the image sensor may be unavoidable, this noise may be reduced using various post-processing methods. The noise reduction improves the qualitative appearance of the image, though at the potential sacrifice of certain details in the image or the generation of various artifacts.
An image of a scene captured by a camera may be visually less appealing to a viewer than the original scene itself due to issues such as incorrect exposure values of the camera when capturing the scene, or the limited dynamic range of the camera. A photosite of an image sensor on a camera may become saturated at a particular exposure setting, and thus any areas of the scene with higher intensities than the saturation point are all shown as the maximum value. On the other hand, there may be a minimal number of photons being detected at the photosite, such that a low value is registered by the photosite that is at the minimum value of the range or below a noise floor. These limitations in the range of intensity values that the camera may capture can lead to images that are blown out, too dark, or a combination of these issues.